SWE-bench Verified is a human-filtered subset of 500 instances from SWE-bench created in collaboration with OpenAI to provide a reliable evaluation set for coding agents and language models. Human annotators reviewed each instance to ensure problem descriptions are clear, test patches are correct, and tasks are solvable with available information. The Verified leaderboard shows results from a wide variety of AI coding systems, and a Bash Only view isolates language model performance using mini-SWE-agent in a minimal bash environment.
- Human validation ensures clarity, correctness of test patches, and solvability
- Bash-only comparison uses mini-SWE-agent with ReAct loop, no tools or scaffold
- Leaderboard distinguishes mini-SWE-agent LM results from full agent systems via Agent dropdown
- Release 2.x uses tool calling vs 1.x parses actions from output strings, making releases not directly comparable
- Human validation ensures clarity, correctness of test patches, and solvability
- Bash-only comparison uses mini-SWE-agent with ReAct loop, no tools or scaffold
- Leaderboard distinguishes mini-SWE-agent LM results from full agent systems via Agent dropdown
- Release 2.x uses tool calling vs 1.x parses actions from output strings, making releases not directly comparable
Iliya Mirzaei writes that test-time methods which make a language model critique, rewrite or reflect on its own output do not beat simple repeated sampling when both are matched on generated tokens. Re-running the budget-matched comparison of Wang et al. as a designed experiment with paired bootstrap intervals and Holm correction on Qwen2.5-1.5B, 3B and 7B for GSM8K and MATH-500 with 150 questions each, no method is reliably better than self-consistency at equal cost; ten comparisons are significantly worse, all involving self-inspection. Best-of-N selection by the model loses to majority voting on identical samples below 7B, while rewriting methods such as Self-Refine and forced Reflexion remain below the equal-cost baseline even at 7B.
- Reflexion as implemented never triggered a retry on the 1.5B model, judging every answer correct and collapsing to a single chain of thought.
- Best-of-N vs counting penalty shrinks with scale but stays negative at 7B; parity is approached, not crossed.
- Cost is generated tokens only; accounting for input tokens would make self-assessing methods look worse.
- All generations, prompts and checking scripts are released for reproducibility.
- Reflexion as implemented never triggered a retry on the 1.5B model, judging every answer correct and collapsing to a single chain of thought.
- Best-of-N vs counting penalty shrinks with scale but stays negative at 7B; parity is approached, not crossed.
- Cost is generated tokens only; accounting for input tokens would make self-assessing methods look worse.
- All generations, prompts and checking scripts are released for reproducibility.
* **Structured Outputs:** Uses grammar-constrained decoding (logit biasing/masking) to enforce strict JSON schema compliance during inference. Best for deterministic data transformation.
* **Function Calling:** Utilizes instruction tuning to enable model reasoning over tool definitions. Best for agentic workflows and external state mutation.
| Feature | Structured Outputs | Function Calling |
| :--- | :--- | :--- |
| **Mechanism** | Constrained decoding (Grammar/Regex) | Instruction-tuned intent detection |
| **Reliability** | 100% Schema Compliance | Probabilistic (requires retry logic) |
| **Primary Use Case** | ETL, Query Gen, Reasoning traces | API Triggers, RAG, Task Routing |
| **Latency/Cost** | Low overhead; optimized decoding | Higher overhead due to tool-definition tokens |
* **ETL & Extraction:** Use Structured Outputs to ensure downstream parsers never fail on malformed JSON.
* **Agentic Loops:** Use Function Calling for multi-turn interactions where the model must decide *which* tool to invoke based on context.
* **Hybrid Pattern (Controller/Formatter):** Deploy a "Function Calling" agent as the **Controller** to select tools, then pipe results through a "Structured Output" layer as the **Formatter** to ensure clean data ingestion into databases or UIs.
* **Function Calling:** Utilizes instruction tuning to enable model reasoning over tool definitions. Best for agentic workflows and external state mutation.
| Feature | Structured Outputs | Function Calling |
| :--- | :--- | :--- |
| **Mechanism** | Constrained decoding (Grammar/Regex) | Instruction-tuned intent detection |
| **Reliability** | 100% Schema Compliance | Probabilistic (requires retry logic) |
| **Primary Use Case** | ETL, Query Gen, Reasoning traces | API Triggers, RAG, Task Routing |
| **Latency/Cost** | Low overhead; optimized decoding | Higher overhead due to tool-definition tokens |
* **ETL & Extraction:** Use Structured Outputs to ensure downstream parsers never fail on malformed JSON.
* **Agentic Loops:** Use Function Calling for multi-turn interactions where the model must decide *which* tool to invoke based on context.
* **Hybrid Pattern (Controller/Formatter):** Deploy a "Function Calling" agent as the **Controller** to select tools, then pipe results through a "Structured Output" layer as the **Formatter** to ensure clean data ingestion into databases or UIs.
This article explains the concept of 'skills' in the context of language models, detailing how to create and use them to enhance model capabilities. It covers the file structure, YAML configuration, and integration of scripts for task automation, providing a practical guide for developers.
We introduce the Ministral 3 series, a family of parameter-efficient dense language models designed for compute and memory constrained applications, available in three model sizes: 3B, 8B, and 14B parameters. For each model size, we release three variants: a pretrained base model for general-purpose use, an instruction finetuned, and a reasoning model for complex problem-solving.
Ollama has partnered with NVIDIA to optimize performance on the new NVIDIA DGX Spark, powered by the GB10 Grace Blackwell Superchip, enabling fast prototyping and running of local language models.
This Perspective outlines ways in which generative artificial intelligence aligns with and supports the core ideas of generative linguistics, and how generative linguistics can provide criteria to evaluate and improve neural language models.
This paper surveys recent replication studies of DeepSeek-R1, focusing on Supervised Fine-Tuning (SFT) and Reinforcement Learning from Verifiable Rewards (RLVR). It details data construction, method design, and training procedures, offering insights and anticipating future research directions for reasoning language models.
An introduction to evaluating language models with easy-to-understand metrics.
Understand temperature, Top-k, Top-p, frequency, and presence penalty for LLM hyperparameters once and for all with visual examples.
